{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'3.8.2 | packaged by conda-forge | (default, Apr 24 2020, 07:34:03) [MSC v.1916 64 bit (AMD64)]'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import sys #先查看一下Python当前的版本\n",
    "sys.version\n",
    "#打印数据"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里我们主要运行第一个爬虫界面\n",
    "使用库：urllib\n",
    "在Python3中摒弃了2里面的urllib2库，转变成urllib.request"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "http.client.HTTPResponse"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import urllib.request\n",
    "response = urllib.request.urlopen('http://www.baidu.com')\n",
    "type(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "我们尝试爬取百度的代码，会得到一个响应。在上述响应中，我们输出它的类型。\n",
    "显示的数据表示是一个HTTPResponse类型对象\n",
    "接下来我们尝试进行相关数数据输出！"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "状态码为： 200\n",
      "获取的头部信息有： Connection: close\n",
      "Content-Length: 48901\n",
      "Server: nginx\n",
      "Content-Type: text/html; charset=utf-8\n",
      "X-Frame-Options: DENY\n",
      "Via: 1.1 vegur\n",
      "Via: 1.1 varnish\n",
      "Accept-Ranges: bytes\n",
      "Date: Sat, 04 Jul 2020 12:09:24 GMT\n",
      "Via: 1.1 varnish\n",
      "Age: 2821\n",
      "X-Served-By: cache-bwi5137-BWI, cache-hkg17934-HKG\n",
      "X-Cache: HIT, HIT\n",
      "X-Cache-Hits: 2, 1486\n",
      "X-Timer: S1593864565.564878,VS0,VE0\n",
      "Vary: Cookie\n",
      "Strict-Transport-Security: max-age=63072000; includeSubDomains\n",
      "\n",
      "\n",
      "头部信息序列组： [('Connection', 'close'), ('Content-Length', '48901'), ('Server', 'nginx'), ('Content-Type', 'text/html; charset=utf-8'), ('X-Frame-Options', 'DENY'), ('Via', '1.1 vegur'), ('Via', '1.1 varnish'), ('Accept-Ranges', 'bytes'), ('Date', 'Sat, 04 Jul 2020 12:09:24 GMT'), ('Via', '1.1 varnish'), ('Age', '2821'), ('X-Served-By', 'cache-bwi5137-BWI, cache-hkg17934-HKG'), ('X-Cache', 'HIT, HIT'), ('X-Cache-Hits', '2, 1486'), ('X-Timer', 'S1593864565.564878,VS0,VE0'), ('Vary', 'Cookie'), ('Strict-Transport-Security', 'max-age=63072000; includeSubDomains')]\n",
      "头部信息是Via的是 1.1 vegur, 1.1 varnish, 1.1 varnish\n",
      "输出读取的url:  \n"
     ]
    }
   ],
   "source": [
    "print(\"状态码为：\",response.status)\n",
    "print(\"获取的头部信息有：\",response.headers)\n",
    "print(\"头部信息序列组：\",response.getheaders())\n",
    "print(\"头部信息是Via的是\",response.getheader(\"Via\"))\n",
    "print(\"输出读取的url: \",response.read().decode('utf-8')) #输出相关的网页内容，返回一段url"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "获取时urlopen的API：\n",
    "response = urllib.request.urlopen(url,data=none,timeout,cafile=none,capath = none,codefault = none,context = none)\n",
    "如果转换成字节流byte 则默认使用Post方法请求，看下面的爬虫模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "pycharm": {
     "is_executing": true,
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b'{\\n  \"args\": {}, \\n  \"data\": \"\", \\n  \"files\": {}, \\n  \"form\": {\\n    \"world\": \"hello\"\\n  }, \\n  \"headers\": {\\n    \"Accept-Encoding\": \"identity\", \\n    \"Content-Length\": \"11\", \\n    \"Content-Type\": \"application/x-www-form-urlencoded\", \\n    \"Host\": \"httpbin.org\", \\n    \"User-Agent\": \"Python-urllib/3.8\", \\n    \"X-Amzn-Trace-Id\": \"Root=1-5f0078a7-f2bfce478bd80f8e50e44972\"\\n  }, \\n  \"json\": null, \\n  \"origin\": \"39.171.15.190\", \\n  \"url\": \"http://httpbin.org/post\"\\n}\\n'\n"
     ]
    }
   ],
   "source": [
    "import urllib.parse\n",
    "data = bytes(urllib.parse.urlencode({'world':'hello'}),encoding='utf-8')\n",
    "response2 = urllib.request.urlopen('http://httpbin.org/post',data = data)\n",
    "print(response2.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上面的代码实现了以表单提交的方式post传输数据\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
